Unsupervised Grounding of Spatial Relations

We present an unsupervised connectionist model for grounding color, shape and spatial relations of two objects in 2D space. The model constitutes a two-layer architecture that integrates information from visual and auditory inputs. The images are presented as the visual inputs to an artificial retina and fiveword sentences describing them (e.g. “Red box above green circle”) serve as auditory inputs with phonological encoding. The visual scene is represented by the Self-Organizing Map(s) and the auditory description is processed by a recursive SOM (RecSOM) that learns to topographically represent sequences. Primary representations are integrated in a multimodal module (implemented by SOM or Neural Gas algorithms) in the second layer using self-organizing units with conjunctive representations. We tested this two-layer architecture in two versions (a single SOM representing color, shape and spatial relations vs. biologically inspired separate SOMs for spatial relations and for shape and color) and several conditions (scenes with varying complexity up to 3 colors, 5 object shapes and 4 spatial relations). In the scenes with higher complexity we reached better results with NG algorithm in the multimodal layer compared to SOM, which is thank to the flexible neighborhood relations in NG algorithm, relaxing topographic organization. The results confirm theoretical assumptions about the different nature of visual and auditory coding. Our model is hence able to efficiently integrate the two sources of information while reflecting their specific features.